Towards a Calibrated Corpus for Compression Testing

A mini-corpus of twelve ‘calibrated’ binary-data files have been produced for systematic evaluation of compression algorithms (Available at http : //www.tcode.auckland.ac.nz/-mark/corpus). These are generated within the framework of a deterministic theory of string complexity [2]. Here the T-complexity of a string z (measured in taugs) is defined as CT(Q) = Ci logz(ki + 1): where the positive integers k, are the T-expansion parameters for the corresponding string production process outlined in [l] . C ( ) T z 1s o bserved in [2] to be the Logarithmic Integral of the total information content Iz, of z (measured in nats), i.e., CT(X) = li(1,). T he average entropy is HZ= I,/jxI, i.e.: the total information content divided by the length of 2. Thus CT(X) = li(H, ~1x1). Alternatively, the information rate along a string may be described by an entropy function Hz(n), 0 < n L: 1x1 for the string [3]. Assuming that Hz(n) is continuously integrable along the length of the 2, then I, = $“’ H,(n)&. T h u s CT(X) = li (JJ”’ H,(n)&). So ving for Hz(n): that is differentiating 1 both sides and rearranging, we get: ~G(4n) Hz(n) = Sn x &I, @-l (w4?J)) (1)